feat(collections): create from and read raw schema JSON - #598
feat(collections): create from and read raw schema JSON#598dudanogueira wants to merge 3 commits into
Conversation
Add a raw-JSON escape hatch to the collections API, in both directions,
on the sync and async clients:
client.collections.createFromJson(json) // POST /schema
client.collections.getConfigAsJson(name) // GET /schema/{name}
client.collections.listAsJson() // GET /schema
Neither direction maps the document onto CollectionConfig. On write the
body is forwarded byte-for-byte, so any option the server accepts works,
including ones this client version does not model yet. On read the
response body is returned untouched.
The read side matters because CollectionConfig deserialization is lossy
today. Most visibly, VectorConfig.CustomTypeAdapterFactory looks for the
bq/pq/sq/rq keys at the top of vectorIndexConfig, which is correct for
hnsw and flat but not for dynamic, where they sit one level deeper under
hnsw/flat. A collection with a dynamic index and RQ enabled reports
quantization = null. Property-level moduleConfig, moduleConfig entries
that are neither reranker-* nor generative-*, and the read-only
shardingConfig fields are dropped as well. getConfigAsJson gives callers
that need the complete picture -- rendering or diffing a schema -- a way
to get it without waiting for the mapping to be fixed.
createFromJson reads exactly one key, "class", which it needs to return
a collection handle. A document that is not a JSON object, or that has
no non-empty string "class", raises IllegalArgumentException before the
request is sent. getConfigAsJson validates nothing and maps 404 to
Optional.empty(), matching the typed getConfig.
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
|
I suggest a different approach, probably one that will result in better code-reuse / less duplication. The only difference between public record GetConfigRequest(String collectionName) {
public static final <T> Endpoint<GetConfigRequest, Optional<T>> endpoint(Class<T> cls) {
return OptionalEndpoint
.<GetConfigRequest, T>noBodyOptional(
request -> "GET",
request -> "/schema/" + request.collectionName,
request -> Collections.emptyMap(),
(statusCode, response) -> JSON.deserialize(response, cls));
}
}This way the same GetConfigRequest can be used to return:
Same for list and create endpoints. Would you like to take a stab at that? I'm happy to take this over, this seems like a small enough change. Wrt to the "Not addressed here" section: if you find any bugs while working on a PR, please open a ticket for those. A PR description is not the right place to document bugs. |
Address review feedback on weaviate#598: the raw-JSON escape hatch no longer needs request classes of its own. The only thing that differed between create(CollectionConfig) and createFromJson(String) was the payload, so the existing request records now take the type as a parameter: GetConfigRequest.endpoint(CollectionConfig.class | String.class) ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class) CreateCollectionRequest<CollectionConfig | String>.endpoint() CreateCollectionFromJsonRequest, GetConfigJsonRequest and ListCollectionJsonRequest are removed; the public client API is unchanged. SimpleEndpoint.deserializeClass special-cases String.class by returning the response body unparsed, so any endpoint parameterized by response type gets the raw option for free. On the write side, a String payload is forwarded byte-for-byte and anything else is serialized first. The create endpoint now returns Void instead of parsing the echoed configuration into CollectionConfig: both clients discarded it, and a raw payload may describe a collection CollectionConfig cannot represent, which would have made that parse throw. The "class" lookup moved to CreateCollectionRequest.collectionNameFromJson, which both clients call before sending, so an unusable document still fails before the request leaves the process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJ9gPAiDmKMN15VyRbw5UR
The server refuses a "dynamic" vector index unless it was started with ASYNC_INDEXING=true, so testCreateFromJsonAndGetConfigAsJson failed with HTTP 422 on every version in CI. Give that test its own container with async indexing enabled rather than flipping the flag on the shared one: async indexing makes the vectors of freshly inserted objects searchable only eventually, and most suites rely on them being searchable at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU
|
Thanks — took a stab at it, done in 94de2ef.
GetConfigRequest.endpoint(CollectionConfig.class | String.class)
ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class)
CreateCollectionRequest.<CollectionConfig | String>endpoint()Rather than a per-request deserializer, Two things I decided along the way, shout if you'd rather have them differently:
On the "Not addressed here" section — fair point, moved out to #606 and #607 and trimmed the description down to a reference. |
What
Adds a raw-JSON escape hatch to the collections API, in both directions, on the sync and async clients:
Neither direction maps the document onto
CollectionConfig. On write the body is forwarded byte-for-byte, so any option the server accepts works — including ones this client version does not model yet. On read the response body is returned untouched.Why the read side is here too
This started as write-only, but the read path turned out to need the same escape hatch:
CollectionConfigdeserialization is lossy today, so there is currently no way to render or diff a real collection faithfully using the client alone (see #606 and #607 for the two parsing bugs found while investigating).Beyond those, also dropped on read: property-level
moduleConfig(the server's actual home forskip/vectorizePropertyName),moduleConfigentries that are neitherreranker-*norgenerative-*, and the read-onlyshardingConfigfields (actualCount,function,key,strategy).getConfigAsJsonunblocks callers who need the complete picture without waiting for the mapping to be fixed.Design notes
The raw option is not a parallel set of request classes:
GetConfigRequest,ListCollectionRequestandCreateCollectionRequestare parameterized by payload type, so the same request serves both the typed and the raw path.SimpleEndpoint.deserializeClassspecial-casesString.classby returning the response body unparsed, so any endpoint parameterized by response type gets the raw option for free. On the write side, aStringpayload is forwarded byte-for-byte and anything else is serialized first.Other decisions worth calling out:
Voidrather than parsing the echoed configuration intoCollectionConfig. Both clients discarded it anyway, and a raw payload may describe a collectionCollectionConfigcannot represent, which would have made that parse throw.CreateCollectionRequest.collectionNameFromJsonreads exactly one key,"class", which the client needs in order to return a collection handle. Both clients call it before sending, so a document that is not a JSON object, or that has no non-empty string"class", raisesIllegalArgumentExceptionbefore the request is sent rather than after a round-trip.getConfigAsJsonvalidates nothing — there is no name to extract — and maps 404 toOptional.empty(), matching the typedgetConfig.create(String)already means "create by collection name", so the new method could not be an overload; hence the distinct names.Testing
createFromJsonvalidation path.CollectionsITestexercising create-from-JSON →getConfigAsJson→listAsJson→ delete, using a dynamic-index-with-RQ payload.Full unit suite passes. The integration tests could not run locally (testcontainers fails to initialize in my environment — all pre-existing ITs error the same way), so the new integration test logic was additionally verified by running an identical copy against a live Weaviate 1.38.0: the collection was created from raw JSON,
getConfigAsJsonreturnedhnsw.rq.enabled = trueintact,listAsJsonsaw it, and the post-delete lookup returned empty. CI will be the real check on the ITs.